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ABSTRACT 

Noting that the use of a postal test for literacy 
measurement is an innovation, this paper describes the four reading 
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starting their studies in the Open University at the foundation 
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A Postal Survey of OU Students’ Reading Skills 



Abstract 

lET’s Text and Readers Programme has tr.sted the reading skills of a sample of 
students starting their studies in the Open University at foundation course level. 

The use of a postal test for literacy measurement is an innovation, though the nature 
of the tests is well established by precedent. It is the first time that large-scale direct 
measures of a basic skill have been taken of Open University students. 

This report describes the tests used in this postal survey. There is a discussion of 
reading skills and their assessment, followed by the rationale for the choice of test 
materials. The survey instrument is reprinted in the Appendix. 



Introduction 

The OU has been operating with great success for 25 years, and routinely collects a 
wide range of data on students’ social, demographic and personal characteristics. 

We also survey student satisfaction with courses, and their opinions on a range of 
study facilities. However, until now, almost no data has been collected on the basic 
skills of our students. 

This project set out to collect data on the skill which seems to be most important for 
our system of learning, the skill of reading. It is generally accepted that for students in 
higher education to be successful they must have (or rapidly acquire) good quality 
reading skills. This is particularly important for OU students, since text material is the 
main part of almost all our courses. Also, there is a wealth of supporting text-based 
material (readers, set books, study guides, and administrative documents and forms). 
When we first considered this project, we knew our students were scattered the length 
and breadth of the UK (and some abroad). It was impractical to use the usual 
methods of psychometric testing, where a psychologist administers tests face-to-face 
with groups or individuals. Instead, we decided to build on our strengths and make 
use of the Institute’s expertise in conducting and analysing postal surveys. 

This is the first time that large scale testing of a basic skill has been carried out ^t the 
Open University. Also, though the tests themselves are of a well-known type, oui use 
of a postal survey for literacy measurement is original. This survey of reading skills is 
part of a larger project which we hope will include an estimation of the readability of 
OU foundation course text materials and of the overall reading workload in the five 
courses. Results of the reading skill survey will be presented in our next report. 

Bernard Scott. Research Fellow 

Michael Macdonald-Ross, Reader in Textual Communication. 





Assessing reading skills 

What are reading skills? Reading is the process of extracting meaning from written 
text. This involves a complex cluster of skills. Much is known about how these skills 
are acquired and interact (Fames 1973; Harri-Augstein, Smith and Thomas 1982; 
Flood 1984; Beech and Colley 1987; Britton and Glynn 1987; Carver 1990). It is usual 
to distinguish lower level skills (e.g. letter and word recognition) from higher level 
skills (e.g. comprehension at the levels of sentence, paragraph and longer passages, the 
use of different strategies for skimming, scanning and reviewing). 

Comprehension at higher levels depends on success at lower levels; looked at this way, 
reading is very much a bottom-up or data-driven process. But in another way reading 
is top-down or concept-driven. Individual words are understood in the context of 
individual sentences; sentences are understood in the context of other sentences; whole 
passages are understood in the context of the reader’s world knowledge and 
assumptions about the writer’s purposes. 

Table 1 is a brief summary of the main factors affecting the comprehension of text. 

Table 1 Factors affecting the comprehension of text: 

Reader'based factors: 

Cognitive: reading and study skills, metacognition, vocabulary, general 
knowledge, sp>ecific knowledge of subject domain and the writer’s intent. 
Motivational: purposes, level of interest. 

Text-based factors: 

Syntactic complexity, semantic complexity, coherence (local and global), 
rhetorical style, legibihty of type. 

How are reading skills measured? The lower level skills of letter and word recognition 
can be isolated and tested in a variety of ways. As long as we can assume basic 
literacy, otir concern as a university is with the higher level skills involved in 
comprehension. Table 2 summaries the main ways in which reading comprehension 
skills are assessed. For reasons described below, we have chosen to use the cloze 
procedure, together with a test that gives an estimate of size of vocabulary. 

Table 2 How to assess reading comprehension skills: 

Read passage and identify deliberate errors. 

Read passage and answer multiple choice questions. 

Read passage and answer open ended questions. 

Read passage and complete practical task. 

Cloze: read passage with words deleted and fill in the gaps. 

Vocabulary tests. 





Why Cloze? 

Cloze was first developed by Taylor (1953). It has been extensively researched both as 
a measure of comprehension skills and as a measure of readability (Bormuth 1966; 
further references in Klare 1984). It correlates very highly with other measures of 
reading comprehension. It is well established that the cloze procedure taps many levels 
of skill, including 

❖ recognition of individual words 

❖ using semantic and syntactic information within a sentence to predict 
a missing word. 

❖ using the local context, semantic and syntactic, of other sentences to 
aid prediction. 

❖ s kimmi ng to recap what has been read and being prepared to revise 
hypotheses about global meanings in the light of new information. 

❖ scanning ahead for cues to aid prediction. 

Unlike other comprehension tests, cloze tests are relatively easy to construct (Bormuth 
1975). They are extremely reliable over repeated testing and scores are relatively robust 
for variable exposure times. It has been shown that simple ‘correct word or not’ scoring 
is as effective at validly distinguishing between subjects as more sophisticated scoring, 
where s 3 monyms for deleted words are accepted as correct or where cues for initial 
letter and/or word length are given or where the missing word is selected from a list of 
alternatives (Miller and Coleman 1967). 

Cloze can also be used as a measure of the readability of texts. In that case texts are 
assessed using a population of readers with known levels of reading skill. Cloze based 
measures have been shown to correlate very highly with other readability measures 
and to give more consistent rankings of text difficulty than traditional tests of reading 
comprehension (Bormuth op.cit.). However, using cloze as a measure of readability is 
a complex and expensive procedure and, in practice, it is far simpler to use other 
measures, some of which are available as software packages. 

The Reading Tests 

We have had to select our own test materials since there are no standardised tests 
suitable for group testing the reading skills of adults (Pumfrey 1985) although there is 
one vocabulary scale with UK norms (Raven, Court and Raven 1994). We have 
included a vocabulary test in our questionnaire but used one more fitting to our 
purposes. For details, see Sources below. 

The passages 

We have selected three passages at three levels of difficulty. The first (and easiest 
passage) should be comprehended by any adult with basic functional literacy. The 
second passage requires more ability, such as, reading middle-brow newspapers and 
novels. The third passage is more difficult; it is a fairly typical example of academic 
' prose, though not by any means the most difficult. 

The difficulty levels of the passages have been assessed and compared using a measure 
of readability, the Flesch Reading Ease Score. Readability scores, such as the Flesch, 
are reliable predictors of the difficulty of text for readers. 

Comprehensibility of text may also usefully be distinguished from the ‘leamability’ of 
text. Instructional materials include many features to encourage effective learning. 
These features may be regarded as ‘add-ons’ since they are extra to the main subject - 
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matter discourse (see Rowntree 1990 for an overview). Table 3 summarises these 
distinctions between readability, comprehensibility and leamability. 

Tables 

Readability: 

Measures of syntactic and semantic complexity that predict the 
difficulty of text comprehension. 

Comprehensibility: 

Accounts of logical/causal/narrative coherence, global and local. 

Leamability: 

Assessment of pedagogical effectiveness of instructional texts 
(comprehensibihty plus use of activities, self-assessment, 
metacommentary on study strategies). 

As noted above, in addition to the three cloze passages, we have also included a 
vocabulary test. From many studies it is known that vocabulary is highly correlated 
with measures of general intelligence. It is also a major factor in readability measures. 
Previous OU work has shown vocabulary to be a good predictor of academic success 
(Woodley and McIntosh 1980). 

The survey instnoment with accompanying letter and instructions is shown in the 
Appendix. 

Sources. 

After deciding that cloze procedure suited our purpose, we set about selecting 
appropriate passages. We reviewed the literature on adult literacy, came across the 
work of ALBSU (the Adult Literacy and Basic Skills Unit) and discovered that 
ALBSU had produced screening materials for reading skills in cloze form. It seemed 
sensible to use their materials to allow for comparisons across the UK adult 
population. 

Passages 1 and 2 are from ALBSU. Passage 1 is part of a basic skills screening test 
used extensively in FE colleges (ALBSU 1993). Scores between 40% and 60% are at 
ALBSU Standards Level 1. broadly equivalent to GNv^Q Communication Skills Level 
1, and National Curriculum English Levels 4-5. 

Passage 2 is from an ALBSU handbook on Cloze procedure, with examples (Vaughan 
1989). According to the author “If a learner can satisfactorily complete one of these 
passages. , , they should be able to read the popular national newspapers and much of 
the reading that is involved in normal everyday life”. 

No ALBSU materials were available at the higher level of difficulty needed for 
Passage 3. so we made our own choice of fairly representative academic prose and 
then developed a cloze version. Passage 3 is based on an extract from a review of an 
academic book which appeared in the Times Higher Education Supplement (Turner 
1994). 

The vocabulary test is taken from a series of tests constructed by the late Hunter 
Diack. The test gives an estimate of vocabulary size. For an account of the test’s 
rationale and construction, see Diack 1975. 

We are most grateful to all the copyright holders for their permissions to use these 
materials in our survey. 
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Appendix: The Survey Instrument 




lET Surveyof Reading Skills & Readability 



Dear Student, 

As part of the OU's efforts to improve the quality of its courses, we are carrying 
out a survey of the readability of our texts and the reading skills of OU students. 
We would be most grateful if you would take part in this exercise. 

To help assess reading skills, we are using a technique known as the Cloze 
procedure. Each of the passages that you are asked to read has had words deleted. 
As you read through the passages you are asked to write down what you think are 
the missing words. The passages are of increasing difficulty, and the third passage 
is especially difficult. Do have a go at it, but do not feel inadequate if its overall 
sense eludes you. 

There is also a short task that gives a rough estimate of the size of your 
vocabulary. We've included a simple scoring procedure for your interest. 

The whole exercise should take about 30 to 45 minutes, although some students 
will take less time than that, and others may take more. There is no time limit; 
please work at your own pace. 

Please note: your performance on the tasks set is strictly confidential to our 
survey. Results are published only in the form of statistical summaries over large 
samples. 

Thank you very much for your cooperation. We wish you all success with your 
studies. 

Yours sincerely. 

The Texts Readers Programme 
Institute of Educational Technology 
Walton Hall 



This survey is not concerned with individual 
admissions, selection or assessment, and is 
not meant for clinical or diagnostic purposes. 
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INSTRUCTIONS 



Here's what we would like you to do: 

1. Find a place where you will be undisturbed. 

2. Start with Part A, and work through it as directed. 

3. Next work through Part B as directed. 

4. When finished, piease return the completed form to Walton Hall. 
The envelope is re-usabie and a prepaid address iabei is provided. 



For this sun/ey to be vaiid, it is vital for you to do the exercises unaided 



iO 
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PartA 

CARD 1 



lET Survey 
of Reading 
Skills 



Instructions 

In this survey are passages in which words have been left out at regular intervals, leaving 
blank spaces. Your task is to fill in each blank space with the word that you think fits best. 

Try this out, using this practice passage. 

Only one word has left out each time this is indicated by a 

continuous line. All the lines of equal length, so they no clue 

as to the of the missing word. Spelling is important, as long as 

it clear what the word that you intended to in the 

gap. 

If at you can’t think of a to put in the gap, read on and 

back to it later. You may also go back and change your choices. 

Now check your answers against the complete passage: 

Only one word has been left out each time and this is indicated by a continuous line. 
All the lines are of equal length, so that they give no clue as to the length of the 
missing word. Spelling is not important, as long as it is clear what the word was that 
you intended to put in the gap. 

If at first you can’t think of a word to put in the gap, read on and go back to it later. 
You may also go back and change your choices. 

Don't worry if some of the words you chose are different from our answers. The important 
thing is that you understand what is being said. 

Note: the passages are of increasing difficulty, and the third passage is quite difficult. 

There is no time limit for the task. You should take about half an hour to finish. 

When ready please begin. 




For office use only 
Please do not writ* 
in this column 

( 1 ) 

(2-5) 

( 6 ) 

(7-14) 



(15-17) 
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First passage: 

Safe as houses 

We think of our home as a safe place to be, and are more worried when someone goes 

out of the house than when they stay at home. In fact people die from 

accidents in . home every year than are on the road or at work. 

are the causes of these in the home? Many people may 

of fire as the greatest , but in fact more people 

from falling than from any cause. Tragic accidents, some fatal 

caused by children and adults , walking or falling through 

glass and windows. Poisoning can also illness or death. This 

may from medicines or from household substances as 

cleaning materials. Food is also a common danger. 

Statistics that most accidents happen on Mondays at 

weekends. People are at work now and have more leisure . 

They therefore spend more time home. This has led to 

increase in the number of in the home. The two vulnerable 

groups of people young children (especially pre-school ) and 

the elderly. Th re groups more time in the home , for example, 

older children who at school, or adults are out at work during the 

• The elderly are weaker and slower reactions. Young children 

are aware of the dangers in home, and are dependent on 

carers being aware of dangerous . 



This passage from ‘Assessing Reading and Maths’ is reproduced by 
kind permission of ALBSU, The Adult Literacy & Basic Skills Unit. 
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( 21 ) 


( 22 ) 


( 23 ) 


( 24 ) 


( 25 ) 


( 26 ) 


( 27 ) 


( 28 ) 


( 29 ) 


( 30 ) 


( 31 ) 


( 32 ) 


( 33 ) 




( 34 ) 


( 35 ) 


( 36 ) 


( 37 ) 


( 38 ) 


( 39 ) 


( 40 ) 


( 41 ) 


( 42 ) 


( 43 ) 


( 44 ) 


( 45 ) 


( 46 ) 


( 47 ) 


( 48 ) 


( 49 ) 


( 50 ) 


( 51 ) 


( 52 ) 


( 53 ) 
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Second passage: 

Travel and tourism 

Work within the tourist industiy involves planning and arranging activities for 
tourists. These holidays have to be sold and a full back-up service provided, to 
meet the customers’ needs. 

Besides holiday travel, business continues to grow each . 

This is an important of income. Air fares getting lower and 

this led to much more travel than ever before. 

The industry may seem glamorous exciting, but the reality 

rather different. There are in exotic holiday spots, 

there are many more involving routine tasks. In this staff 

are needed for clerical sales work in offices travel agencies. 

Junior staff may reductions on holidays occasionally have 

training trips . Mainly it is management who go overseas on 

trips. Couriers and representatives abroad, but many of 

are employed on a temporary . If you want to as 

a courier or , being able to speak local language will be 

. In some other jobs language may be useful, 

not essential. In many a language will not used at all. 

Tour arrange the transport, acconunodation leisure 

activities that make a holiday package. Travel act like a 

link the client and the providers, tour operators. 

Tourist boards promote in their country. They research into 

current needs and future trends in tourism. 

Guides usually have a specialised knowledge of one area and provide a service to 
visitors. They may take guided tours round an area or an important building such 
as a cathedral or stately home. 

This passage from ‘Assessing Reading’ by Judy Vaughan is reproduced 
by kind permission of ALBSU, The Adult Literacy & Basic Skills Unit. 
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Third passage 



Rethinking university teaching 

The reduction of the unit of resource for in institutions of higher 

turns our attention to possibility of increasing resource-based 

• The opportunities which educational offers for maintaining 

quality higher education look ever attractive. A book which 

the possibilities of different , and sets out a within 

which media can evaluated, is indeed timely. Diana Laurillard 

this in her book. Rethinking University Teaching, eulogy, and without 

suggesting technology offers a panacea. book which argues 

elegantly a sound basis in research into student learning 

welcome. She gives concrete of good practice, without 

letting her book become “how to do it” . This book 

will offer to anybody who is with the development of 

centres in higher education. 

book is in three . In the first, Laurillard out to 

characterise the which are essential to in a university. 

Academic (and the qualifying adjective important) is presented 

as conversation. There is a of exposition, in which 

teacher sets out his her description of the . Students are then 

encouraged ' set out their descriptions the world. On the 

that the two differ, teacher then identifies tasks the 

student is to , selected to highlight the between the two 

descriptions, to draw attention to weaknesses of the student’s 

• The teacher and student reformulate their descriptions in 

light of the discourse, the process can continue. 
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This passage is from 'The importance of the Reflective Classes’ by David Turner, THES 29.4.94. 
Its use here is authorised by kind permission of the author and Times Newspapers Ltd. 
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Parts 

A rough estimate of the size of your vocabuiary 

Start at word numbered (2), read the words in order. Put a yin the box for each word you know. 

For each of the last five of your ticked words, please show that you do know its meaning. 

To do this you give a definition, show in a sentence how it is used, or make a small sketch. 

Card 3 



application 


□ 


(2) 


almoner D 


(22) 


benefit 


□ 


(3) 


bibliography D 


(23) 


crater 


□ 


(4) 


cognate D 


(24) 


demonstration 


□ 


(5) 


daguerrotype D 


(25) 


factor 


□ 


(6) 


homonym D 


(26) 


hitch 


□ 


(7) 


largesse D 


(27) 


jury 


□ 


(8) 


malign □ 


(28) 


loiter 


□ 


(9) 


minuend D 


(29) 


mildew 


□ 


(10) 


neap D 


(30) 


ordeal 


□ 


(11) 


penultimate D 


(31) 


abdicate 


□ 


(12) 


avatar D 


(32) 


catastrophe 


□ 


(13) 


cheval-glass D 


(33) 


impresario 


□ 


(14) 


demiurge □ 


(34) 


irreparable 


□ 


(15) 


farandole D 


(35) 


linear 


□ 


(16) 


homocentric D 


(36) 


mercantile 


□ 


(17) 


kaolin D 


(37) 


nitrogen 


□ 


(18) 


leat D 


(38) 


officious 


□ 


(19) 


Uanero D 


(39) 


perspective 


□ 


(20) 


marram D 


(40) 


radiant 


□ 


(21) 


onager D 


(41) 



List the last five words you ticked: And for each one give its meaning: 

WORD MEANING 



A rough estimate of vocabulary size is given by the number of ticks plus 10 multiplied by 600. 
For example, if you have 20 ticks, vocabulary size is: (20 + 10)X600 = 30 X 600 = 18000 words. 



This word-list is adapted from ‘Standard Literacy Tests' by Hunter Diack. 

Its use here is authorised by kind permission of the trustees of the late Mr Diack's estate. 
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Thank you very much for your help! 
Please use the pre^paid label and return 
this completed form as soon as possible to: 

The Survey Office 
Institute of Educational Technology 
The Open University 
PO Box 175 

Milton Keynes MK7 SAW 





14 



Text & Readers Programme 
Publications 



Technical Reports 

1 : The revolution in print technology 
2: A postal survey of OU students’ reading skills 
3: Results of the survey of OU students’ reading skills 
4: OU students’ reading skills and final registration 

Newsflash 

1: OU students’ reading skills 



o 



17 



